Soft touchless sensors and touchless sensing for soft robots.

Soft robots are characterized by their mechanical compliance, making them well-suited for various bio-inspired applications. However, the challenge of preserving their flexibility during deployment has necessitated using soft sensors which can enhance their mobility, energy efficiency, and spatial adaptability. Through emulating the structure, strategies, and working principles of human senses, soft robots can detect stimuli without direct contact with soft touchless sensors and tactile stimuli. This has resulted in noteworthy progress within the field of soft robotics. Nevertheless, soft, touchless sensors offer the advantage of non-invasive sensing and gripping without the drawbacks linked to physical contact. Consequently, the popularity of soft touchless sensors has grown in recent years, as they facilitate intuitive and safe interactions with humans, other robots, and the surrounding environment. This review explores the emerging confluence of touchless sensing and soft robotics, outlining a roadmap for deployable soft robots to achieve human-level dexterity.

Multi-silicone bilateral soft physical twin as an alternative to traditional user interfaces for remote palpation: a comparative study.

Teleoperated medical technologies are a fundamental part of the healthcare system. From telemedicine to remote surgery, they allow remote diagnosis and treatment. However, the absence of any interface able to effectively reproduce the sense of touch and interaction with the patient prevents the implementation of teleoperated systems for primary care examinations, such as palpation. In this paper, we propose the first reported case of a soft robotic bilateral physical twin for remote palpation. By creating an entirely soft interface that can be used both to control the robot and receive feedback, the proposed device allows the user to achieve remote palpation by simply palpating the soft physical twin. This is achieved through a compact design showcasing 9 pneumatic chambers and exploiting multi-silicone casting to minimize cross-noise and allow teleoperation. A comparative study has been run against a traditional setup, and both the control and feedback of the physical twin are carefully analyzed. Despite distributed tactile feedback not achieving the same performance as the visual map, the soft control and visual feedback combination showcases a 5.1% higher accuracy. Moreover, the bilateral soft physical twin results always in a less invasive procedure, with 41% lower mechanical work exchanged with the remote phantom.

Variable sensitivity multimaterial robotic e-skin combining electronic and ionic conductivity using electrical impedance tomography.

Electronic skins (e-skins) aim to replicate the capabilities of human skin by integrating electronic components and advanced materials into a flexible, thin, and stretchable substrate. Electrical impedance tomography (EIT) has recently been adopted in the area of e-skin thanks to its robustness and simplicity of fabrication compared to previous methods. However, the most common EIT configurations have limitations in terms of low sensitivities in areas far from the electrodes. Here we combine two piezoresistive materials with different conductivities and charge carriers, creating anisotropy in the sensitive part of the e-skin. The bottom layer consists of an ionically conducting hydrogel, while the top layer is a self-healing composite that conducts electrons through a percolating carbon black network. By changing the pattern of the top layer, the resulting distribution of currents in the e-skin can be tuned to locally adapt the sensitivity. This approach can be used to biomimetically adjust the sensitivities of different regions of the skin. It was demonstrated how the sensitivity increased by 500% and the localization error reduced by 40% compared to the homogeneous case, eliminating the lower sensitivity regions. This principle enables integrating the various sensing capabilities of our skins into complex 3D geometries. In addition, both layers of the developed e-skin have self-healing capabilities, showing no statistically significant difference in localization performance before the damage and after healing. The self-healing bilayer e-skin could recover full sensing capabilities after healing of severe damage.

Coordinating upper limbs for octave playing on the piano via neuro-musculoskeletal modeling.

Understanding the coordination of multiple biomechanical degrees of freedom in biological organisms is crucial for unraveling the neurophysiological control of sophisticated motor tasks. This study focuses on the cooperative behavior of upper-limb motor movements in the context of octave playing on the piano. While the vertebrate locomotor system has been extensively investigated, the coherence and precision timing of rhythmic movements in the upper-limb system remain incompletely understood. Inspired by the spinal cord neuronal circuits (central pattern generator, CPG), a computational neuro-musculoskeletal model is proposed to explore the coordination of upper-limb motor movements during octave playing across varying tempos and volumes. The proposed model incorporates a CPG-based nervous system, a physiologically-informed mechanical body, and a piano environment to mimic human joint coordination and expressiveness. The model integrates neural rhythm generation, spinal reflex circuits, and biomechanical muscle dynamics while considering piano playing quality and energy expenditure. Based on real-world human subject experiments, the model has been refined to study tempo transitions and volume control during piano playing. This computational approach offers insights into the neurophysiological basis of upper-limb motor coordination in piano playing and its relation to expressive features.

Sensorized Skin With Biomimetic Tactility Features Based on Artificial Cross-Talk of Bimodal Resistive Sensory Inputs.

Tactility in biological organisms is a faculty that relies on a variety of specialized receptors. The bimodal sensorized skin, featured in this study, combines soft resistive composites that attribute the skin with mechano- and thermoreceptive capabilities. Mimicking the position of the different natural receptors in different depths of the skin layers, a multi-layer arrangement of the soft resistive composites is achieved. However, the magnitude of the signal response and the localization ability of the stimulus change with lighter presses of the bimodal skin. Hence, a learning-based approach is employed that can help achieve predictions about the stimulus using 4500 probes. Similar to the cognitive functions in the human brain, the cross-talk of sensory information between the two types of sensory information allows the learning architecture to make more accurate predictions of localization, depth, and temperature of the stimulus contiguously. Localization accuracies of 1.8 mm, depth errors of 0.22 mm, and temperature errors of 8.2 °C using 8 mechanoreceptive and 8 thermoreceptive sensing elements are achieved for the smaller inter-element distances. Combining the bimodal sensing multilayer skins with the neural network learning approach brings the artificial tactile interface one step closer to imitating the sensory capabilities of biological skin.

Vocal pain expression augmentation for a robopatient.

Abdominal palpation is one of the basic but important physical examination methods used by physicians. Visual, auditory, and haptic feedback from the patients are known to be the main sources of feedback they use in the diagnosis. However, learning to interpret this feedback and making accurate diagnosis require several years of training. Many abdominal palpation training simulators have been proposed to date, but very limited attempts have been reported in integrating vocal pain expressions into physical abdominal palpation simulators. Here, we present a vocal pain expression augmentation for a robopatient. The proposed robopatient is capable of providing real-time facial and vocal pain expressions based on the exerted palpation force and position on the abdominal phantom of the robopatient. A pilot study is conducted to test the proposed system, and we show the potential of integrating vocal pain expressions to the robopatient. The platform has also been tested by two clinical experts with prior experience in abdominal palpation. Their evaluations on functionality and suggestions for improvements are presented. We highlight the advantages of the proposed robopatient with real-time vocal and facial pain expressions as a controllable simulator platform for abdominal palpation training studies. Finally, we discuss the limitations of the proposed approach and suggest several future directions for improvements.

On the Stability and Behavioral Diversity of Single and Collective Bernoulli Balls.

The ability to express diverse behaviors is a key requirement for most biological systems. Underpinning behavioral diversity in the natural world is the embodied interaction between the brain, body, and environment. Dynamical systems form the basis of embodied agents, and can express complex behavioral modalities without any conventional computation. While significant study has focused on designing dynamical systems agents with complex behaviors, for example, passive walking, there is still a limited understanding about how to drive diversity in the behavior of such systems. In this article, we present a novel hardware platform for studying the emergence of individual and collective behavioral diversity in a dynamical system. The platform is based on the so-called Bernoulli ball, an elegant fluid dynamics phenomenon in which spherical objects self-stabilize and hover in an airflow. We demonstrate how behavioral diversity can be induced in the case of a single hovering ball via modulation of the environment. We then show how more diverse behaviors are triggered by having multiple hovering balls in the same airflow. We discuss this in the context of embodied intelligence and open-ended evolution, suggesting that the system exhibits a rudimentary form of evolutionary dynamics in which balls compete for favorable regions of the environment and exhibit intrinsic "alive" and "dead" states based on their positions in or outside of the airflow.

Robotic Simulators for Tissue Examination Training With Multimodal Sensory Feedback.

Tissue examination by hand remains an essential technique in clinical practice. The effective application depends on skills in sensorimotor coordination, mainly involving haptic, visual, and auditory feedback. The skills clinicians have to learn can be as subtle as regulating finger pressure with breathing, choosing palpation action, monitoring involuntary facial and vocal expressions in response to palpation, and using pain expressions both as a source of information and as a constraint on physical examination. Patient simulators can provide a safe learning platform to novice physicians before trying real patients. This paper reviews state-of-the-art medical simulators for the training for the first time with a consideration of providing multimodal feedback to learn as many manual examination techniques as possible. The study summarizes current advances in tissue examination training devices simulating different medical conditions and providing different types of feedback modalities. Opportunities with the development of pain expression, tissue modeling, actuation, and sensing are also analyzed to support the future design of effective tissue examination simulators.

Sensing, Actuating, and Interacting Through Passive Body Dynamics: A Framework for Soft Robotic Hand Design.

Robotic hands have long strived to reach the performance of human hands. The physical complexity and extraordinary capabilities of the human hand, in terms of sensing, actuation, and cognitive abilities, make achieving this goal challenging. At the heart of the physical structure of the hand is its' passive behaviors. Seen most clearly in soft robotic hands, these behaviors influence and affect the mechanical, sensing, and control functionalities. With this perspective, we present a framework through which passivity in robot hands can be understood, by concretely identifying the role of passivity in the design, fabrication, and control of soft hands. In this framework we focus on the interactions between the physical hand and the: environment, internal actuation, sensor morphology, and wrist control. Taking these surrounding systems away, we are left with a passive soft hand whose behaviors emerge from external interactions. Inspired by the human hand, we define the role of these four key interacting pillars and review how state-of-the art robot hands utilize these four elements to aid functionality. We show how these pillars promote hybrid soft-rigid hands with rich behaviors, providing benefits in terms of the increased adaptability to uncertain environments, improved scalability and reduction in the cost of actuation, sensing, and control. This review provides a conceptual framework for approaching hand design and analysis through consideration of the passive behaviors. This highlights not only the advances that can be made by approaching the problem in this way but also the outstanding challenges that stem from this outlook.

Energy Harvesting for Robots with Adaptive Morphology.

Robots primarily made of soft and elastic materials have potential applications such as traveling in confined spaces due to their adaptive morphology. However, their energy efficiency is still subject to improvement. Although a possible approach to increase efficiency is by harvesting the energy used during their behavioral motion, it is not trivial to do so due to their complex dynamics. This work seeks to pioneer a study that exploits the tight coupling between a robot's adaptive morphology, control, and consequent behaviors to harvest energy and increase energy efficiency. It is hypothesized that since varying the robot's morphology may change the energy use that leads to contrasting behavior and efficiency, harvesting the robot's energy will need to be adapted to its morphology. To verify the hypothesis, we developed a shape-changing robot with an elastic structure that achieves locomotion via vibration controlled by a single motor, such that the complex dynamics of the robot can be characterized through its resonance frequencies. It will be shown that harvesting energy at opportune occasions is more important than maximizing the harvest capacity to increase energy efficiency. We will also show how the robot's shape affects energy use in locomotion and how energy harvesting will feedback additional energy that increases the magnitude and affects the robot's behavior. We conclude with an understanding of the role of the robot's morphology, that is, shape, in using the energy provided to the robot and how the understanding can be used to harvest the robot's energy to increase its efficiency.

Autonomous detection and sorting of litter using deep learning and soft robotic grippers.

Road infrastructure is one of the most vital assets of any country. Keeping the road infrastructure clean and unpolluted is important for ensuring road safety and reducing environmental risk. However, roadside litter picking is an extremely laborious, expensive, monotonous and hazardous task. Automating the process would save taxpayers money and reduce the risk for road users and the maintenance crew. This work presents LitterBot, an autonomous robotic system capable of detecting, localizing and classifying common roadside litter. We use a learning-based object detection and segmentation algorithm trained on the TACO dataset for identifying and classifying garbage. We develop a robust modular manipulation framework by using soft robotic grippers and a real-time visual-servoing strategy. This enables the manipulator to pick up objects of variable sizes and shapes even in dynamic environments. The robot achieves greater than 80% classified picking and binning success rates for all experiments; which was validated on a wide variety of test litter objects in static single and cluttered configurations and with dynamically moving test objects. Our results showcase how a deep model trained on an online dataset can be deployed in real-world applications with high accuracy by the appropriate design of a control framework around it.

Data-driven method for damage localization on soft robotic grippers based on motion dynamics.

Damage detection is one of the critical challenges in operating soft robots in an industrial setting. In repetitive tasks, even a small cut or fatigue can propagate to large damage ceasing the complete operation process. Although research has shown that damage detection can be performed through an embedded sensor network, this approach leads to complicated sensorized systems with additional wiring and equipment, made using complex fabrication processes and often compromising the flexibility of the soft robotic body. Alternatively, in this paper, we proposed a non-invasive approach for damage detection and localization on soft grippers. The essential idea is to track changes in non-linear dynamics of a gripper due to possible damage, where minor changes in material and morphology lead to large differences in the force and torque feedback over time. To test this concept, we developed a classification model based on a bidirectional long short-time memory (biLSTM) network that discovers patterns of dynamics changes in force and torque signals measured at the mounting point. To evaluate this model, we employed a two-fingered Fin Ray gripper and collected data for 43 damage configurations. The experimental results show nearly perfect damage detection accuracy and 97% of its localization. We have also tested the effect of the gripper orientation and the length of time-series data. By shaking the gripper with an optimal roll angle, the localization accuracy can exceed 95% and increase further with additional gripper orientations. The results also show that two periods of the gripper oscillation, i.e., roughly 50 data points, are enough to achieve a reasonable level of damage localization.

State-transfer modeling collective behavior of multi-ball Bernoulli system based on local interaction forces.

Collective behavior observed in nature has been actively employed in swarm robotics. In order to better respond to external cues, the agents in such systems organize themselves in an ordered structure based on simple local rules. The central assumption, in swarm robotics, is that all agents in the system collaborate to fulfill a common goal. In nature, however, many multi-agent systems exhibit a more complex collective behavior involving a certain level of competition. One representative example of complex collective behavior is a multi-ball Bernoulli-ball system. In this paper, by extracting local force among the Bernoulli balls, we approximated the state-transfer model mapping interaction forces to observed behaviors. The results show that the collective Bernoulli-ball system spent 41% of its time on competitive behaviors, in which up to 84% of the interaction state is unorganized. The rest 59% of the time is spent on collaborative behavior. We believe that the novel proposed model opens new avenues in swarm robotics research.

3D Printable Soft Sensory Fiber Networks for Robust and Complex Tactile Sensing.

The human tactile system is composed of multi-functional mechanoreceptors distributed in an optimized manner. Having the ability to design and optimize multi-modal soft sensory systems can further enhance the capabilities of current soft robotic systems. This work presents a complete framework for the fabrication of soft sensory fiber networks for contact localization, using pellet-based 3D printing of piezoresistive elastomers to manufacture flexible sensory networks with precise and repeatable performances. Given a desirable soft sensor property, our methodology can design and fabricate optimized sensor morphologies without human intervention. Extensive simulation and experimental studies are performed on two printed networks, comparing a baseline network to one optimized via an existing information theory based approach. Machine learning is used for contact localization based on the sensor responses. The sensor responses match simulations with tunable performances and good localization accuracy, even in the presence of damage and nonlinear material properties. The potential of the networks to function as capacitive sensors is also demonstrated.

How the Environment Shapes Tactile Sensing: Understanding the Relationship Between Tactile Filters and Surrounding Environment.

The mechanical properties of a sensor strongly affect its tactile sensing capabilities. By exploiting tactile filters, mechanical structures between the sensing unit and the environment, it is possible to tune the interaction dynamics with the surrounding environment. But how can we design a good tactile filter? Previously, the role of filters' geometry and stiffness on the quality of the tactile data has been the subject of several studies, both implementing static filters and adaptable filters. State-of-the-art works on online adaptive stiffness highlight a crucial role of the filters' mechanical behavior in the structure of the recorded tactile data. However, the relationship between the filter's and the environment's characteristics is still largely unknown. We want to show the effect of the environment's mechanical properties on the structure of the acquired tactile data and the performance of a classification task while testing a wide range of static tactile filters. Moreover, we fabricated the filters using four materials commonly exploited in soft robotics, to merge the gap between tactile sensing and robotic applications. We collected data from the interaction with a standard set of twelve objects of different materials, shapes, and textures, and we analyzed the effect of the filter's material on the structure of such data and the performance of nine common machine learning classifiers, both considering the overall test set and the three individual subsets made by all objects of the same material. We showed that depending on the material of the test objects, there is a drastic change in the performance of the four tested filters, and that the filter that matches the mechanical properties of the environment always outperforms the others.

Face mediated human-robot interaction for remote medical examination.

Realtime visual feedback from consequences of actions is useful for future safety-critical human-robot interaction applications such as remote physical examination of patients. Given multiple formats to present visual feedback, using face as feedback for mediating human-robot interaction in remote examination remains understudied. Here we describe a face mediated human-robot interaction approach for remote palpation. It builds upon a robodoctor-robopatient platform where user can palpate on the robopatient to remotely control the robodoctor to diagnose a patient. A tactile sensor array mounted on the end effector of the robodoctor measures the haptic response of the patient under diagnosis and transfers it to the robopatient to render pain facial expressions in response to palpation forces. We compare this approach against a direct presentation of tactile sensor data in a visual tactile map. As feedback, the former has the advantage of recruiting advanced human capabilities to decode expressions on a human face whereas the later has the advantage of being able to present details such as intensity and spatial information of palpation. In a user study, we compare these two approaches in a teleoperated palpation task to find the hard nodule embedded in the remote abdominal phantom. We show that the face mediated human-robot interaction approach leads to statistically significant improvements in localizing the hard nodule without compromising the nodule position estimation time. We highlight the inherent power of facial expressions as communicative signals to enhance the utility and effectiveness of human-robot interaction in remote medical examinations.

Reach Space Analysis of Baseline Differential Extrinsic Plasticity Control.

The neuroplasticity rule Differential Extrinsic Plasticity (DEP) has been studied in the context of goal-free simulated agents, producing realistic-looking, environmentally-aware behaviors, but no successful control mechanism has yet been implemented for intentional behavior. The goal of this paper is to determine if "short-circuited DEP," a simpler, open-loop variant can generate desired trajectories in a robot arm. DEP dynamics, both transient and limit cycles are poorly understood. Experiments were performed to elucidate these dynamics and test the ability of a robot to leverage these dynamics for target reaching and circular motions.

Mastication-Enhanced Taste-Based Classification of Multi-Ingredient Dishes for Robotic Cooking.

Chefs frequently rely on their taste to assess the content and flavor of dishes during cooking. While tasting the food, the mastication process also provides continuous feedback by exposing the taste receptors to food at various stages of chewing. Since different ingredients of the dish undergo specific changes during chewing, the mastication helps to understand the food content. The current methods of electronic tasting, on the contrary, always use a single taste snapshot of a homogenized sample. We propose a robotic setup that uses the mixing to imitate mastication and tastes the dish at two different mastication phases. Each tasting is done using a conductance probe measuring conductance at multiple, spatially distributed points. This data is used to classify 9 varieties of scrambled eggs with tomatoes. We test four different tasting methods and analyze the resulting classification performance, showing a significant improvement over tasting homogenized samples. The experimental results show that tasting at two states of mechanical processing of the food increased classification F1 score to 0.93 in comparison to the traditional tasting of a homogenized sample resulting in F1 score of 0.55. We attribute this performance increase to the fact that different dishes are affected differently by the mixing process, and have different spatial distributions of the salinity. It helps the robot to distinguish between dishes of the same average salinity, but different content of ingredients. This work demonstrates that mastication plays an important role in robotic tasting and implementing it can improve the tasting ability of robotic chefs.

Closing the Control Loop with Time-Variant Embedded Soft Sensors and Recurrent Neural Networks.

Embedded soft sensors can significantly impact the design and control of soft-bodied robots. Although there have been considerable advances in technology behind these novel sensing materials, their application in real-world tasks, especially in closed-loop control tasks, has been severely limited. This is mainly because of the challenge involved with modeling a nonlinear time-variant sensor embedded in a complex soft-bodied system. This article presents a learning-based approach for closed-loop force control with embedded soft sensors and recurrent neural networks (RNNs). We present learning protocols for training a class of RNNs called long short-term memory (LSTM) that allows us to develop accurate and robust state estimation models of these complex dynamical systems within a short period of time. Using this model, we develop a simple feedback force controller for a soft anthropomorphic finger even with significant drift and hysteresis in our feedback signal. Simulation and experimental studies are conducted to analyze the capabilities and generalizability of the control architecture. Experimentally, we are able to develop a closed-loop controller with a control frequency of 25 Hz and an average accuracy of 0.17 N. Our results indicate that current soft sensing technologies can already be used in real-world applications with the aid of machine learning techniques and an appropriate training methodology.